Evaluating the Predictive Accuracy of Association Football Forecasting Systems

Despite the increasing importance and popularity of association football forecasting systems there is no agreed method of evaluating their accuracy. We have classified the evaluators used into two broad categories: those which consider only the prediction for the observed outcome; and those which consider the predictions for the unobserved as well as observed outcome. We highlight fundamental inconsistencies between them and demonstrate that they produce wildly different conclusions about the accuracy of four different forecasting systems (Fink Tank/Castrol Predictor, Bet365, Odds Wizard, and pi-football) based on recent Premier league data. None of the existing evaluators satisfy a set of simple theoretical benchmark criteria. Hence, it is dangerous to assume that any existing evaluator can adequately assess the performance of football forecasting systems and, until evaluators are developed that address all the benchmark criteria, it is best to use multiple types of predictive evaluators (preferably based on posterior validation).